• Introduction
  • Research synthesis
  • Comparison of meta-analysis and BES
  • Meta-analysis
  • Bayesian evidence synthesis
• Simulation
  • Data generation
  • Sampling population effect sizes
  • Hypotheses
  • Simulation results I
  • Simulation results II

• Many conflicting results in psychological science (OSC, 2015)
  
  
• Further exacerbated by small sample sizes and reliance on statistical significance tests (Button et al., 2013; van Calster et al., 2018)
  
  
• Increasing the robustness of results by quantitatively summarizing multiple studies
  • Meta-analysis
  • Bayesian Evidence Synthesis (BES)

Computing meta-analytic effect size \(\bar{\beta}\) and its variance \(v\) (Hedges & Vevea 1998):

\[ \begin{aligned} \bar{\beta} &= \frac{\sum_{k=1}^{K} w_k\hat{\beta_k}}{\sum_{k=1}^{K} w_k} \\ \\ w_k &= \frac{1}{var(\hat{\beta_k})} \\ \\ v &= \frac{1}{\sum_{k=1}^{K} w_k} \end{aligned} \]
where \(K\) is the number of studies, and \(\hat{\beta_k}\) is the estimated effect size of study \(k\).

• Combine studies at the level of the hypothesis instead of the effect size
• Compute study-specific Bayes factors and turn these into posterior model probabilities (PMPs)
• Then compute an aggregated PMP over all studies (Kuiper et al., 2012):

\[ \pi_{K,h} = \frac{\prod_{k=1}^{K}\pi_{k,h}}{\sum_{h=1}^{H}\prod_{k=1}^{K}\pi_{k,h}} \]
where \(h\) is hypothesis 1, …, \(H\) and \(\pi\) is the PMP.

Generate data from a regression model with one predictor: \[ Y = \beta X_1 + \epsilon, \] where \(X_1\) is sampled from \(\mathcal{N}(0,1)\) and \(\epsilon\) is sampled from from \(\mathcal{N}(0,\sqrt{1-R^2})\).

beta <- 0.3                                                    # specify beta value
R2 <- beta^2                                                   # R-squared
n <- 25                                                        # sample size
X <- rnorm(n = n, mean = 0, sd = 1)                            # sample X values
Y <- X * beta + rnorm(n = n, mean = 0, sd = sqrt(1 - R2))      # compute Y

Instead of assuming one underlying population effect size, we assume study-specific population effect sizes that randomly deviate from the overall mean following a normal distribution \(\mathcal{N}(0.3,0.1)\):

We tested \(H1: \beta > 0\) against either the null hypothesis (\(H0\)), the complement hypothesis (\(Hc\)) or the unconstrained hypothesis (\(Hu\)):

\[ \begin{aligned} H0&: \ \beta = 0 \\ Hc&: \ \beta < 0 \\ Hu&: \ \beta \end{aligned} \]
where \(Hu\) means that \(\beta\) can take on any value.

Proportion of simulations (nsim = 1000) that H1 gets selected over the alternative hypothesis based on the aggregated PMP:

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376. https://doi.org/10.1038/nrn3502

Hedges, L. V., & Vevea, J. L. (1998). Fixed-and random-effects models in meta-analysis. Psychological Methods, 3(4), 486. https://doi.org/10.1037/1082-989X.3.4.486

Kuiper, R. M., Buskens, V., Raub, W., & Hoijtink, H. (2013). Combining statistical evidence from several studies: A method using Bayesian updating and an example from research on trust problems in social and economic exchange. Sociological Methods & Research, 42(1), 60-81. https://doi.org/10.1177/0049124112464867

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

Van Calster, B., Steyerberg, E. W., Collins, G. S., & Smits, T. (2018). Consequences of relying on statistical significance: Some illustrations. European Journal of Clinical Investigation, 48(5), e12912. https://doi.org/10.1111/eci.12912